CATASTROPHIC FORGETTING

MoExtend: Tuning New Experts for Modality and Task Extension (2024.acl-srw)

Challenge:	Existing models for Query-product relevance classification are not accurate across multiple languages.
Approach:	They propose a multilingual continual learning framework that adds adapters for each new language and incorporates a fusion layer above language-specific adapters.
Outcome:	The proposed approach reduces trainable parameters by 80% while outperforming SOTA CL methods on proprietary and external datasets.

Copied to clipboard

Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, Pan Zhou

Federated Incremental Named Entity Recognition (2025.coling-main)

Challenge:	Existing instruction tuning methods for large language models (LLMs) are costly and difficult to implement.
Approach:	They propose a framework to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models.
Outcome:	The proposed framework enables rapid adaptation and extension to new modal data or tasks without tuning pretrained models.

Copied to clipboard

Zesheng Liu, Qiannan Zhu, Cuiping Li, Hong Chen

Dual Contrastive Learning Framework for Incremental Text Classification (2023.findings-emnlp)

Challenge:	Existing methods for named entity recognition are based on pre-fixed entity types, resulting in catastrophic forgetting.
Approach:	They propose a model which allows for catastrophic forgetting of old entity types . they propose adaptive pseudo labeling and a prototypical relation distillation loss .
Outcome:	The proposed model overcomes catastrophic forgetting problem on old entity types with semantic shift.

Copied to clipboard

Yigong Wang, Zhuoyi Wang, Yu Lin, Jinghui Guo, Sadaf Halim, Latifur Khan

Fine-tuning Large Language Models with Limited Data: A Survey and Practical Guide (2026.tacl-1)

Challenge:	In incremental learning, large models learn and refresh knowledge continuously . many approaches have been proposed to preserve knowledge from previous tasks while learning new concepts in online NLP applications.
Approach:	They propose a dual contrastive learning framework that fosters transferability across different tasks . they use global contrastive and task-specific learning to promote a generalized embedding space .
Outcome:	The proposed framework outperforms the current state-of-the-art methods on text datasets.

Copied to clipboard

Marton Szep, Daniel Rueckert, Rüdiger von Eisenhart-Rothe, Florian Hinterwimmer

Refining Sample Embeddings with Relation Prototypes to Enhance Continual Relation Extraction (2021.acl-long)

Challenge:	Pre-trained language models provide strong foundations, but effective adaptation under data scarcity requires efficient and efficient fine-tuning techniques.
Approach:	They propose to review parameter-efficient fine-tuning techniques that lower training and deployment costs and domain and cross-lingual adaptation methods for both encoder and decoder models.
Outcome:	The proposed techniques lower training and deployment costs, domain and cross-lingual adaptation methods, and model specialization strategies.

Copied to clipboard

Li Cui, Deqing Yang, Jiaxin Yu, Chengwei Hu, Jiayang Cheng, Jingjie Yi, Yanghua Xiao

How Relevant is Selective Memory Population in Lifelong Language Learning? (2022.aacl-short)

Challenge:	Existing methods to extract relationships from texts depend on memory size and replay these memorized samples in subsequent tasks.
Approach:	They propose to use a model to extract relations between entities from texts where the samples of different relations are delivered into the model continuously.
Outcome:	The proposed model outperforms the state-of-the-art models and avoids catastrophic forgetting.

Copied to clipboard

Vladimir Araujo, Helena Balabin, Julio Hurtado, Alvaro Soto, Marie-Francine Moens

Challenge:	Existing approaches to lifelong language learning rely on sparse experience replay to prevent catastrophic forgetting.
Approach:	They propose to use a selective memory population to store a uniform number of samples from the entire data stream to improve model performance.
Outcome:	The proposed methods show that they are relevant for lifelong language learning tasks, especially for low memory size, and consistent with computer vision studies.

Continual Few-shot Intent Detection (2022.coling-1)

Copied to clipboard

Guodun Li, Yuchen Zhai, Qianglong Chen, Xing Gao, Ji Zhang, Yin Zhang

Sentiment Analysis

Challenge:	Existing intent detection systems are trained with lots of labeled data over a predefined set of intent classes.
Approach:	They propose a prefix-guided lightweight encoder with three auxiliary strategies to prevent catastrophic forgetting and negative knowledge transfer across tasks.
Outcome:	The proposed system prevents catastrophic forgetting and encourages positive knowledge transfer across tasks.

CLLE: A Benchmark for Continual Language Learning Evaluation in Multilingual Machine Translation (2022.findings-emnlp)

Copied to clipboard

Han Zhang, Sheng Zhang, Yang Xiang, Bin Liang, Jinsong Su, Zhongjian Miao, Hui Wang, Ruifeng Xu

Exclusive Supermask Subnetwork Training for Continual Learning (2023.findings-acl)

Challenge:	Existing benchmarks for Continual Language Learning (CLL) are limited due to the complexity of the task and the lack of unified benchmarks.
Approach:	They propose a Continual Language Learning Evaluation benchmark CLLE in multilingual translation.
Outcome:	The proposed method is effective when compared with other strong benchmarks.

Copied to clipboard

Prateek Yadav, Mohit Bansal

Sparse Adapter Fusion for Continual Learning in NLP (2026.eacl-long)

Challenge:	Continual Learning (CL) methods focus on accumulating knowledge over time while preventing catastrophic forgetting.
Approach:	They propose a CL method that finds a supermask for each new task that keeps or removes each weight to produce a subnetwork.
Outcome:	The proposed method outperforms strong previous methods on NLP and Vision domains while preventing forgetting.

Copied to clipboard

Min Zeng, Xi Chen, Haiqin Yang, Yike Guo

InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions (2024.naacl-long)

Challenge:	Existing methods for learning continuously face challenges such as inefficient parameter reuse across tasks and catastrophic forgetting when tasks are dissimilar.
Approach:	They propose a Sparse Adapter Fusion Method which dynamically fuses old and new adapters to address these challenges.
Outcome:	The proposed method outperforms state-of-the-art methods while utilizing less than 60% of the parameters.

Copied to clipboard

Yifan Wang, Yafei Liu, Chufan Shi, Haoling Li, Chen Chen, Haonan Lu, Yujiu Yang

Rehearsal-Free Modular and Compositional Continual Learning for Language Models (2024.naacl-short)

Challenge:	In order to perform downstream tasks, Large Language Models (LLMs) need continual adaptation without catastrophic forgetting.
Approach:	They propose a new paradigm that allows for continual adaptation without catastrophic forgetting . they propose to replay previous data based on task similarity with instructions .
Outcome:	The proposed method improves performance over 16 tasks with different training orders.

Copied to clipboard

Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, Hinrich Schuetze

Efficient Meta Lifelong-Learning with Limited Memory (2020.emnlp-main)

Challenge:	Existing methods to overcome catastrophic forgetting are rehearsal-based and parameter isolation-based.
Approach:	They propose a rehearsal-free framework which continuously adds new modules to language models and composes them with existing modules.
Outcome:	Experiments on benchmarks show that MoCL outperforms state-of-the-art and effectively facilitates knowledge transfer.

Copied to clipboard

Zirui Wang, Sanket Vaibhav Mehta, Barnabas Poczos, Jaime Carbonell

AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2021.eacl-main)

Challenge:	Existing natural language learning models fail to continuously learn new tasks as they are re-trained throughout their lifetime.
Approach:	They propose a meta-lifelong framework that combines three common lifelong learning principles . they propose to store past examples in episodic memory and replay them at training and inference time .
Outcome:	The proposed framework achieves state-of-the-art performance using 1% memory size and narrows the gap with multi-task learning.

Copied to clipboard

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych

Few-Shot Class-Incremental Learning for Named Entity Recognition (2022.acl-long)

Challenge:	Existing methods for incorporating knowledge from multiple tasks suffer from catastrophic forgetting and difficulties in dataset balancing.
Approach:	They propose an algorithm that extracts and combine adapters in a knowledge composition step.
Outcome:	The proposed class outperforms traditional methods such as full fine-tuning and multi-task learning on 16 diverse NLU tasks.

Copied to clipboard

Rui Wang, Tong Yu, Handong Zhao, Sungchul Kim, Subrata Mitra, Ruiyi Zhang, Ricardo Henao

Incremental Sequence Labeling: A Tale of Two Shifts (2024.findings-acl)

Challenge:	Existing models of Named Entity Recognition (NER) are trained on large datasets with predefined entity classes, but data of new classes arrives constantly. Existing work on NER relies on the assumption that there exists abundance of labeled data for the training of new class.
Approach:	They propose a few-shot class-incremental learning problem where NER model is trained with only few labeled samples of the new classes without forgetting knowledge of the old ones.
Outcome:	The proposed model improves over existing baselines by reconstructing training data of old classes and real data from the training set.

Copied to clipboard

Shengjie Qiu, Junhao Zheng, Zhen Liu, Yicheng Luo, Qianli Ma

Overcoming Catastrophic Forgetting in Massively Multilingual Continual Learning (2023.findings-acl)

Challenge:	Existing approaches to incremental sequence labeling have focused on the E2O and O2E issues, but neglect the O2e issue.
Approach:	They propose a framework for incremental sequence labeling without semantic shifts that mitigate catastrophic forgetting in models by using knowledge distillation to maintain the model’s discriminative ability for old entities.
Outcome:	The proposed framework mitigates catastrophic forgetting in models while maintaining discriminative ability for old entities while minimizing the model’s bias towards new entities.

Copied to clipboard

Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, Daniel Preotiuc-Pietro

Generative Replay Inspired by Hippocampal Memory Indexing for Continual Language Learning (2023.eacl-main)

Challenge:	Existing methods to handle catastrophic forgetting fail to retain knowledge learnt in the past when sudden shifts occur in training data distributions.
Approach:	They propose a learning rate scheduling method that preserves new information without strongly overwriting past knowledge.
Outcome:	The proposed method preserves new information without overwriting past knowledge in a multilingual continuous learning framework.

Copied to clipboard

Aru Maekawa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura

Flexible Weight Tuning and Weight Fusion Strategies for Continual Named Entity Recognition (2024.findings-acl)

Challenge:	Continual learning (CL) is a fundamental requirement for human-like general intelligence (Parisi et al., 2019).
Approach:	They propose to control sample generation using compressed features of previous training samples by using hippocampal memory indexing to enhance the generative replay.
Outcome:	The proposed method outperforms current generative replay methods and generates training samples from previous tasks.

Copied to clipboard

Yahan Yu, Duzhen Zhang, Xiuyi Chen, Chenhui Chu

Prompt Augmented Generative Replay via Supervised Contrastive Learning for Lifelong Intent Detection (2022.findings-naacl)

Challenge:	Existing methods for Named Entity Recognition (CNER) use knowledge distillation to retain old knowledge, but they are too expensive and fail to integrate with existing state-of-the-art models.
Approach:	They propose a weight tuning and weightfusion strategy to learn new entity types while mitigating catastrophic forgetting of old models.
Outcome:	The proposed strategies improve the performance of existing models and are model-agnostic.

Copied to clipboard

Vaibhav Varshney, Mayur Patidar, Rajat Kumar, Lovekesh Vig, Gautam Shroff

Dynamic Sampling Strategies for Multi-Task Reading Comprehension (2020.acl-main)

Challenge:	Existing methods to identify all possible user intents at design time are expensive and require storage of past data.
Approach:	They propose to continually train an intent detector on new intents while maintaining performance on prior intents.
Outcome:	The proposed method outperforms exemplar replay-based approaches on lifelong intent detection tasks and achieves state-of-the-art on four public datasets.

Copied to clipboard

Ananth Gottumukkala, Dheeru Dua, Sameer Singh, Matt Gardner

L4: Mutual Learning Helps Lifelong Language Learning (2025.emnlp-industry)

Challenge:	Prior work focused on model architecture or generalization to held out datasets and largely passed over the particulars of the multi-task learning set up.
Approach:	They propose a dynamic sampling strategy that selects instances proportional to the model's current performance on a dataset relative to its single task performance.
Outcome:	The proposed model outperforms the best model on ORB, a recent multitask reading comprehension benchmark.

Copied to clipboard

Jiyong Li, Dilshod Azizov, Shangsong Liang

Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting (2023.acl-short)

Challenge:	Existing distillation methods rely on domain-specific teachers, limiting their ability to update in real-time and adapt to dynamic environments.
Approach:	They propose a framework that enables continuous mutual learning from task streams without relying on domain-specific teachers.
Outcome:	The proposed framework reduces catastrophic forgetting while improving performance on various benchmark datasets making it suitable for real-world, dynamic natural language processing (NLP) applications.

Copied to clipboard

Zahra Fatemi, Chen Xing, Wenhao Liu, Caimming Xiong

FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing (2024.naacl-long)

Challenge:	Existing studies addressing gender bias of pre-trained language models, usually build a small gender-neutral data set and conduct a second phase pre-training with such data.
Approach:	They propose a method to improve gender fairness of pre-trained models with less forgetting by evaluating them with general NLP tasks in GLUE.
Outcome:	The proposed method improves gender fairness of pre-trained models with less forgetting and performs better on GLUE by a large margin.

Copied to clipboard

Chen Liu, Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych

LRQuant: Learnable and Robust Post-Training Quantization for Large Language Models (2024.acl-long)

Challenge:	Standard fine-tuning of language models suffers with generalization to distribution shifts.
Approach:	They propose to use Fisher Information to investigate scheduled unfreezing algorithms for adapter-based cross-lingual task transfer to improve generalization to distribution shifts.
Outcome:	The proposed method achieves an average of 2 points improvement over four datasets compared to standard fine-tuning and provides empirical evidence for a theory-based justification of the proposed method.

Copied to clipboard

Jiaqi Zhao, Miao Zhang, Chao Zeng, Ming Wang, Xuebo Liu, Liqiang Nie

Robustness in NLP

Challenge:	Existing methods for post-training quantization (PTQ) are limited by the complexity of the quantization parameter and performance degradations when tested on unseen datasets.
Approach:	They propose a learnable smooth-based PTQ framework that allows for rapid adaptation during testing.
Outcome:	The proposed framework improves performance on unseen datasets and reduces memory constraints.

Enhancing Discriminative Representation in Similar Relation Clusters for Few-Shot Continual Relation Extraction (2025.naacl-long)

Copied to clipboard

Anh Duc Le, Nam Le Hai, Thanh Xuan Nguyen, Linh Ngo Van, Nguyen Thi Ngoc Diep, Sang Dinh, Thien Huu Nguyen

Cross-Attention is All You Need: Adapting Pretrained Transformers for Machine Translation (2021.emnlp-main)

Challenge:	Existing methods for relation extraction (RE) fail to address the problem of similar relations, which contributes to catastrophic forgetting.
Approach:	They propose a relation extraction method that utilizes relation descriptions and dynamic clustering to identify similar relations.
Outcome:	The proposed method mitigates catastrophic forgetting and outperforms state-of-the-art methods by a large margin.

Copied to clipboard

Mozhdeh Gheini, Xiang Ren, Jonathan May

Overcoming Catastrophic Forgetting beyond Continual Learning: Balanced Training for Neural Machine Translation (2022.acl-long)

Challenge:	a series of experiments show that fine-tuning only the cross-attention parameters is nearly as effective as fine-timing all parameters.
Approach:	They conduct experiments to fine-tune a translation model on data where either the source or target language has changed.
Outcome:	The proposed model can be trained to several new languages with reduced parameter storage overhead.

Copied to clipboard

Chenze Shao, Yang Feng

MTIVE: Multi-Task Image Verification Engine Using Vision-Language Models for E-commerce (2026.acl-industry)

Challenge:	Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions.
Approach:	They propose a method that iteratively provides complementary knowledge to student models by dynamically updating teacher models trained on specific data orders.
Outcome:	The proposed method improves on multiple machine translation tasks and improves performance over baseline systems.

Copied to clipboard

Yu-Tong Cao, Vishnu Prabhakaran, Arunita Das, Purav Aggarwal, Anoop Saladi

Role Prompting Guided Domain Adaptation with General Capability Preserve for Large Language Models (2024.findings-naacl)

Challenge:	Vision-language models struggle with noisy real-world images and multi-task requirements.
Approach:	They propose a curriculum learning framework that adapts vision-language models through three stages . MTIVE uses frozen base weights with stacked LoRA adapters for shared domain knowledge .
Outcome:	MTIVE outperforms open-source and proprietary baselines in standard and continual learning settings.

Copied to clipboard

Rui Wang, Fei Mi, Yi Chen, Boyang Xue, Hongru Wang, Qi Zhu, Kam-Fai Wong, Ruifeng Xu

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models (2024.emnlp-main)

Challenge:	Large Language Models (LLMs) suffer catastrophic forgetting when tailored to specific domains . authors present a novel approach to manage multi-domain LLM adaptation .
Approach:	They propose a strategy to manage multi-domain LLM adaptation using self-distillation and role integration.
Outcome:	The proposed model alleviates catastrophic forgetting and inter-domain confusion while maintaining robust general capabilities.

Copied to clipboard

Renzhi Wang, Piji Li

Improving Continual Relation Extraction through Prototypical Contrastive Learning (2022.coling-1)

Challenge:	Existing methods for single and batch model editing fail to apply or perform sub-optimally when faced with lifelong model editing.
Approach:	They propose an advanced Mixture of Experts (MoE) adaptor for lifelong model editing that incorporates a novel KV anchor routing method to enhance routing consistency between training and inference stage.
Outcome:	The proposed method surpasses existing models while maintaining outstanding performance in batch editing task.

Copied to clipboard

Chengwei Hu, Deqing Yang, Haoliang Jin, Zhen Chen, Yanghua Xiao

An Evaluation of Progressive Neural Networksfor Transfer Learning in Natural Language Processing (2020.lrec-1)

Challenge:	Continual relation extraction (CRE) aims to extract relations towards the continuous and iterative arrival of new data, of which the major challenge is the catastrophic forgetting of old tasks.
Approach:	They propose a Continual Relation Extraction framework with Contrastive Learning which is built with a classification network and a prototypical contrastive network to achieve incremental-class learning of CRE.
Outcome:	The proposed framework outperforms the state-of-the-art methods on two public datasets and proves its effectiveness on improving performance.

Copied to clipboard

Abdul Moeed, Gerhard Hagerer, Sumit Dugar, Sarthak Gupta, Mainak Ghosh, Hannah Danner, Oliver Mitevski, Andreas Nawroth, Georg Groh

Domain-Lifelong Learning for Dialogue State Tracking via Knowledge Preservation Networks (2021.emnlp-main)

Challenge:	Fine-tuning suffers from catastrophic forgetting, a problem exacerbated in natural language processing (NLP).
Approach:	They propose to use progressive neural networks to re-use previously learned knowledge when learning new tasks.
Outcome:	The proposed approach improves on common NLP tasks across a range of architectures, datasets, and tasks.

Copied to clipboard

Qingbin Liu, Pengfei Cao, Cao Liu, Jiansong Chen, Xunliang Cai, Fan Yang, Shizhu He, Kang Liu, Jun Zhao

Dialogue Systems & Conversational Agents

Challenge:	Existing offline DST models require a fixed dataset to train . Existing domain-lifelong learning methods are impractical in real-world applications .
Approach:	They propose a domain-lifelong learning method to continuously train a DST model on new data to learn incessantly emerging new domains while avoiding catastrophically forgetting old learned domains.
Outcome:	The proposed method outperforms state-of-the-art lifelong learning methods by 4.25% and 8.27% on the MultiWOZ and the SGD benchmarks.

Lifelong Learning of Hate Speech Classification on Social Media (2021.naacl-main)

Copied to clipboard

Jing Qian, Hong Wang, Mai ElSherief, Xifeng Yan

Text Classification

Challenge:	Existing work on automated hate speech classification assumes that the dataset is fixed and the classes are pre-defined.
Approach:	They propose to use Variational Representation Learning and a load-balancing self-organizing inductive neural network to learn hate speech classification on social media.
Outcome:	The proposed model improves on the lifelong learning techniques on social media.

Non-Parametric Adaptation for Neural Machine Translation (N19-1)

Copied to clipboard

Ankur Bapna, Orhan Firat

SharpSeq: Empowering Continual Event Detection through Sharpness-Aware Sequential-task Learning (2024.naacl-long)

Challenge:	Neural Networks trained with gradient descent are susceptible to catastrophic forgetting due to parameter shift during the training process.
Approach:	They propose a semi-parametric approach that relies on local phrase level similarities to retrieve neighboring phrases that are useful for translation even when overall sentence similarity is low.
Outcome:	The proposed approach performs well on a heterogeneous dataset with WMT, IWSLT, JRC-Acquis and OpenSubtitles.

Copied to clipboard

Thanh-Thien Le, Viet Dao, Linh Nguyen, Thi-Nhung Nguyen, Linh Ngo, Thien Nguyen

Cross-Domain Classification of Moral Values (2022.findings-naacl)

Challenge:	Existing methods for continual event detection suffer from catastrophic forgetting . a novel continual learning paradigm leveraging sharpness-aware minimization is needed .
Approach:	They propose a continual learning paradigm that leverages sharpness-aware minimization and a generative model to balance training data distribution.
Outcome:	The proposed approach outperforms existing methods on real-world datasets.

Copied to clipboard

Enrico Liscio, Alin Dondera, Andrei Geadau, Catholijn Jonker, Pradeep Murukannaiah

Text Classification

Challenge:	Existing methods to identify moral values in text can be challenging for transferring knowledge between domains.
Approach:	They compare a deep learning model with a domain-specific value classifier to find out whether it can transfer knowledge to new domains.
Outcome:	The proposed model can generalize and transfer knowledge to novel domains, but introduce catastrophic forgetting.

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (N19-1)

Copied to clipboard

Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos

QAEncoder: Towards Aligned Representation Learning in Question Answering Systems (2025.acl-long)

Challenge:	Existing transfer learning methods employ language models pretrained on large generic corpora, but results come at a high computational cost and require task-specific architectures.
Approach:	They propose a transfer learning approach that combine a task-specific optimization function with an auxiliary language model objective, which is adjusted during the training process.
Outcome:	The proposed method surpasses well established transfer learning methods with greater level of complexity on a variety of affective and text classification tasks surpassing well established methods with higher level of difficulty.

Copied to clipboard

Zhengren Wang, Qinhan Yu, Shida Wei, Zhiyu Li, Feiyu Xiong, Xiaoxing Wang, Simin Niu, Hao Liang, Wentao Zhang

Empowering Large Language Model for Continual Video Question Answering with Collaborative Prompting (2024.emnlp-main)

Challenge:	Modern QA systems entail retrieval-augmented generation (RAG) for accurate and trustworthy responses, but the inherent gap between user queries and relevant documents hinders precise matching.
Approach:	They propose a retrieval-augmented generation (RAG)-based approach to bridge this gap by attaching document fingerprints to the embedding to estimate the expectation of potential queries.
Outcome:	Experiments across diverse datasets, languages, and embedding models confirm the proposed solution is simple-yet-effective with zero additional index storage, retrieval latency, training costs, or catastrophic forgetting and hallucination issues.

Copied to clipboard

Chen Cai, Zheng Wang, Jianjun Gao, Wenyang Liu, Ye Lu, Runzhong Zhang, Kim-Hui Yap

Rational LAMOL: A Rationale-based Lifelong Learning Framework (2021.acl-long)

Challenge:	Existing VideoQA models struggle to adapt to new questions or tasks posed by newly available content.
Approach:	They propose a continual learning framework that fine-tunes a large language model for a sequence of tasks and integrates specific question constraint prompting, knowledge acquisition prompting and visual temporal awareness prompting.
Outcome:	The proposed model achieves 55.14% accuracy on both NExT-QA and DramaQA datasets and 71.24% accuracy for DramaQA.

Copied to clipboard

Kasidis Kanwatchara, Thanapapas Horsuwan, Piyawat Lertvittayakumjorn, Boonserm Kijsirikul, Peerapon Vateekul

Distilling Causal Effect from Miscellaneous Other-Class for Continual Named Entity Recognition (2022.emnlp-main)

Challenge:	Existing paradigms for machine learning suffer from catastrophic forgetting when a model completely forgets what it just learned in previous tasks.
Approach:	They propose to exploit unsupervised rationale generation to improve the performance of a lifelong language learning model by applying critical freezing guided by human rationales.
Outcome:	The proposed framework outperforms vanilla LAMOL on most permutations and unsupervised rationale generation consistently improves the overall performance.

Copied to clipboard

Junhao Zheng, Zhanxian Liang, Haibin Chen, Qianli Ma

CORES: Code-Oriented Reasoning for Complex Text-to-SQL and Generalizable TableQA (2026.findings-acl)

Challenge:	Existing methods for Named Entity Recognition (NER) are not able to learn Other-Class in the same way as new entity types.
Approach:	They propose a unified causal framework to retrieve causality from new entity types and Other-Class.
Outcome:	The proposed method outperforms the state-of-the-art method on three benchmark datasets.

Copied to clipboard

Meng Zhang, Ruochun Jin, Yuanxi Peng, Wenjing Yang, Haotian Wang, Liting Sun, Kun Hu, Silin Yang, Zhang Ke-di

Continued Pretraining and Interpretability-Based Evaluation for Low-Resource Languages: A Galician Case Study (2025.findings-acl)

Challenge:	Text-to-SQL models struggle with complex analytical tasks such as generating simple SQL queries.
Approach:	They propose a text-to-sql model that leverages Python as a procedural reasoning pivot to enhance both complex SQL generation and tabular reasoning.
Outcome:	The proposed model outperforms baseline models on six text-to-SQL benchmarks by 6.44% on average while maintaining good capability on three tableQA benchmarks.

Copied to clipboard

Pablo Rodríguez, Silvia Paniagua Suárez, Pablo Gamallo, Susana Sotelo Docio

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture (2025.naacl-long)

Challenge:	Recent advances in large language models have led to remarkable improvements in language understanding and text generation.
Approach:	They propose a framework to evaluate large language models for underrepresented languages . they examine CPT strategies for languages with limited representation in multilingual models .
Outcome:	The proposed evaluation framework is based on the case of Galician language . it assesses trade-offs between linguistic enrichment and task-solving capabilities .

Copied to clipboard

Jiayi Han, Liang Du, Hongwei Du, Xiangguo Zhou, Yiwen Wu, Yuanfang Zhang, Weibo Zheng, Donghong Han

ZmBART: An Unsupervised Cross-lingual Transfer Framework for Language Generation (2021.findings-acl)

Challenge:	balancing the training budget, downstream performance, and general capabilities of large language models remains a challenge in many applications.
Approach:	They propose a mixture of expert framework based on Soft LoRA and Identity Mixture . SLIM allows dynamic routing between LoRA adapters and identity layers .
Outcome:	The proposed framework reduces training cost while maintaining general capabilities . it can be open-sourced upon publication.

Copied to clipboard

Kaushal Kumar Maurya, Maunendra Sankar Desarkar, Yoshinobu Kano, Kumari Deepshikha

Completely Modular Fine-tuning for Dynamic Language Adaptation (2026.findings-eacl)

Challenge:	Recent advances in NLP focus on large annotated training data.
Approach:	They propose an unsupervised framework that does not use parallel or pseudo-parallel/back-translated data.
Outcome:	The proposed framework does not use parallel or pseudo-parallel/back-translated data.

Copied to clipboard

Zhe Cao, Yusuke Oda, Qianying Liu, Akiko Aizawa, Taro Watanabe

Knowledge Graph-Driven Memory Editing with Directional Interventions (2025.findings-emnlp)

Challenge:	Existing studies on multilingual fine-tuning with a fixed set of languages lack dynamic adaptability to new languages.
Approach:	They propose a modular fine-tuning pipeline that enables dynamic language adaptation for LLMs by first training English-centric adapters for each language separately and then merging them for arbitrary-direction translation.
Outcome:	The proposed pipeline achieves 86% performance over traditional fine-tuning on four languages, while training only 0.1% parameters and relying on English as a bridge language without catastrophic forgetting.

Copied to clipboard

Jinhu Fu, Kun Wang, Chongye Guo, Junfeng Fang, Wentao Zhang, Sen Su

Is Parameter Collision Hindering Continual Learning in LLMs? (2025.coling-main)

Challenge:	Large Language Models (LLMs) are hampered by inaccuracies and outdated information.
Approach:	They propose a framework that constructs knowledge graphs using available information to guide the direction of knowledge editing.
Outcome:	The proposed framework allows consistent, aligned, and stable information during large-scale editing scenarios.

Copied to clipboard

Shuo Yang, Kun-Peng Ning, Yu-Yang Liu, Jia-Yu Yao, Yong-Hong Tian, Yi-Bing Song, Li Yuan

Semi-Supervised Lifelong Language Learning (2022.findings-emnlp)

Challenge:	Existing methods to learn multiple tasks in parallel often lead to catastrophic forgetting, resulting in overwriting knowledge.
Approach:	They propose a non-collision low-rank Adaptation approach that leverages low collision rates to enhance continual learning (CL) in large language models.
Outcome:	The proposed approach achieves better task orthogonality and higher task orthognality than existing SOTA methods.

Copied to clipboard

Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, Nevin L. Zhang

Continual-learning for Modelling Low-Resource Languages from Large Language Models (2026.eacl-long)

Challenge:	Existing methods to learn languages only focus on supervised learning, and unlabeled data is underexplored.
Approach:	They propose a semi-supervised lifelong language learning setting where a model learns sequentially arriving language tasks with both labeled and unlabeled data.
Outcome:	The proposed model outperforms baseline models on various language tasks and is effective and superior to existing models.

Copied to clipboard

Santosh Srinath K, Mudit Somani, Varun Reddy Padala, Prajna Devi Upadhyay, Abhijit Das

Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation (2021.naacl-main)

Challenge:	Existing models for low-resource languages with catastrophic forgetting pose several challenges, including learning to model multi-lingual scenarios.
Approach:	They propose to employ a continual learning strategy using parts-of-speech code-switching and replay adapter strategies to mitigate catastrophic forgetting gap while training LLM from LLM.
Outcome:	The proposed architecture is able to train LLMs from LLM and mitigate catastrophic forgetting gap on vision language tasks.

Copied to clipboard

Shuhao Gu, Yang Feng, Wanying Xie

Continual Learning for Neural Machine Translation (2021.naacl-main)

Challenge:	Existing methods for domain adaptation suffer from catastrophic forgetting, large domain divergence, and model explosion.
Approach:	They propose a method which prunes the model and keeps the important neurons or parameters responsible for both general-domain and in-domain translation.
Outcome:	The proposed method improves on different language pairs and domains compared with strong baselines.

Copied to clipboard

Yue Cao, Hao-Ran Wei, Boxing Chen, Xiaojun Wan

Distill and Replay for Continual Language Learning (2020.coling-main)

Challenge:	Neural machine translation models are data-driven and require large-scale training corpus . continual learning remains a big challenge for artificial intelligence systems and models .
Approach:	They propose a continual learning framework for NMT models that incorporates multiple stages of training to alleviate catastrophic forgetting problem.
Outcome:	The proposed framework achieves superior performance compared to baseline models in all settings.

Copied to clipboard

Jingyuan Sun, Shaonan Wang, Jiajun Zhang, Chengqing Zong

Systematic Knowledge Injection into Large Language Models via Diverse Augmentation for Domain-Specific RAG (2025.findings-naacl)

Challenge:	Existing models fail to isolate acquired knowledge and forget previously learned tasks when learning in a stream where data distribution may shift.
Approach:	They propose a framework that distills knowledge and replays experience from previous tasks when fitting on a new task.
Outcome:	The proposed framework outperforms state-of-the-art models in continuously learning tasks of the same type but from different domains, as well as tasks of different types.

Copied to clipboard

Kushagra Bhushan, Yatin Nandwani, Dinesh Khandelwal, Sonam Gupta, Gaurav Pandey, Dinesh Raghu, Sachindra Joshi

Unveiling Linguistic Regions in Large Language Models (2024.acl-long)

Challenge:	Retrieval-Augmented Generation (RAG) enhances response relevance by incorporating retrieved domain knowledge in the context, retrieval errors can still lead to hallucinations and incorrect answers.
Approach:	They propose a framework that augments the learning process by context augmentation and knowledge paraphrasing by incorporating retrieved domain knowledge into the context.
Outcome:	The proposed framework achieves 10% relative gain in token-level recall while preserving the LLM’s generalization capabilities.

Copied to clipboard

Zhihao Zhang, Jun Zhao, Qi Zhang, Tao Gui, Xuanjing Huang

Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering (P19-1)

Challenge:	Existing studies on how LLMs achieve cross-lingual alignment and generalization have not explored the intrinsic mechanisms of how they achieve crosslingual alignment.
Approach:	They propose to remove a core region that corresponds to linguistic competence and set parameters to zero to reduce performance across 30 different languages.
Outcome:	The proposed model can be used to perform tasks requiring abstract knowledge and reasoning in complex languages.

Copied to clipboard

Claudio Greco, Barbara Plank, Raquel Fernández, Raffaella Bernardi

Visual Data in NLP

Challenge:	Existing methods to overcome catastrophic forgetting in visual question answering models are inadequate, but have received little attention within natural language processing.
Approach:	They devise a set of linguistically-informed visual question answering tasks motivated by psycholinguistics and investigate impact of task difficulty on continual learning.
Outcome:	The proposed models differ in the types of questions they ask and show that task difficulty and order matter.

Hey, That’s My Data! Token-Only Dataset Inference in Large Language Models (2026.findings-acl)

Copied to clipboard

Chen Xiong, Zihao Wang, Rui Zhu, Tsung-Yi Ho, Pin-Yu Chen, Jingwei Xiong, Haixu Tang

kNN For Whisper And Its Effect On Bias And Speaker Adaptation (2025.findings-naacl)

Challenge:	Existing dataset inference methods require logit access, but many modern LLMs restrict such access.
Approach:	They propose a token-only dataset inference framework that allows models to overwrite prior knowledge when trained on new data.
Outcome:	The proposed framework overwrites prior knowledge when trained on new data.

Copied to clipboard

Maya K. Nachesa, Vlad Niculae

DAdEE: Unsupervised Domain Adaptation in Early Exit PLMs (2024.findings-emnlp)

Challenge:	Speech recognition performance varies by language, domain, speaker characteristics such as accent . fine-tuning models on any of these categories may lead to catastrophic forgetting .
Approach:	They propose a nonparametric method that adapts to any category without training the model.
Outcome:	The proposed method improves the performance of a transformer end-to-end speech model on transcribed audio in over 680,000 hours of training data on a wide variety of languages.

Copied to clipboard

Divya Jyoti Bajpai, Manjesh Hanawal

Unlocking Continual Learning Abilities in Language Models (2024.findings-emnlp)

Challenge:	Pre-trained Language Models (PLMs) exhibit good accuracy and generalization ability but their large size results in high inference latency.
Approach:	They propose an unsupervised domain adaptation framework that employs knowledge distillation to achieve domain-invariant representations at each layer.
Outcome:	The proposed framework outperforms early exit methods and domain adaptation methods under domain shift scenarios.

Copied to clipboard

Wenyu Du, Shuang Cheng, Tongxu Luo, Zihan Qiu, Zeyu Huang, Ka Chun Cheung, Reynold Cheng, Jie Fu

SEE: Continual Fine-tuning with Sequential Ensemble of Experts (2025.findings-acl)

Challenge:	Existing approaches to learning models (LMs) incorporate old task data or task-wise inductive bias into LMs, but old data and accurate task information are often unavailable or costly to collect.
Approach:	They propose a rehearsal-free method that updates model parameters with large magnitudes . they found that the L1-normalized magnitude distribution is different when different task data is used .
Outcome:	The proposed method improves accuracy and performance on four CL benchmarks.

Copied to clipboard

Zhilin Wang, Yafu Li, Xiaoye Qu, Yu Cheng

An Empirical Investigation Towards Efficient Multi-Domain Language Model Pre-training (2020.emnlp-main)

Challenge:	Continual fine-tuning of large language models suffers from catastrophic forgetting . some approaches use routers to assign tasks to experts, but continual learning often requires retraining .
Approach:	They propose a framework that integrates routing and response mechanisms within each expert . it eliminates the need for an additional router and allows each expert to decide whether a query should be handled .
Outcome:	The proposed framework outperforms previous approaches in continual fine-tuning . it can handle learning tasks and out-of-distribution instances, paving the way for distributed model ensembling.

Copied to clipboard

Kristjan Arumae, Qing Sun, Parminder Bhatia

Overcoming Catastrophic Forgetting During Domain Adaptation of Seq2seq Language Generation (2022.naacl-main)

Challenge:	Pre-training large language models is a standard practice in the natural language processing community.
Approach:	They propose to use elastic weight consolidation to mitigate catastrophic forgetting when pre-trained large language models are evaluated on generic benchmarks.
Outcome:	The proposed model achieves state-of-the-art on out-of domain tasks with minimal pre-training . elastic weight consolidation provides best overall scores yielding only a 0.33% drop in performance across seven generic tasks while remaining competitive in bio-medical tasks.

Copied to clipboard

Dingcheng Li, Zheng Chen, Eunah Cho, Jie Hao, Xiaohu Liu, Fan Xing, Chenlei Guo, Yang Liu

Balancing Knowledge Breadth and Task Depth for Effective Domain Adaptation Fine-Tuning (2026.findings-acl)

Challenge:	Existing work on lifelong learning requires incremental memory space to learn a model . existing work on experience replay or elastic weighted consolidation requires incremental space .
Approach:	They propose a framework that leverages a recall optimization mechanism to memorize parameters of previous tasks via regularization and a domain drift estimation algorithm to compensate the drift between different domains in the embedding space.
Outcome:	The proposed framework outperforms SOTA models on paraphrase and dialog response generation tasks.

Copied to clipboard

Mu Zhang, Yuxiang Chu, Guangya Yu, Yongqi Fan, Weiyan Zhang, Hang Hu, Tong Ruan, Jingping Liu

Ask Question First for Enhancing Lifelong Language Learning (2022.coling-1)

Challenge:	a lack of knowledge breadth and task depth can hinder curriculum learning in domains such as medicine and finance.
Approach:	They propose a two-dimensional curriculum learning framework that coordinates model training along two orthogonal axes: the knowledge dimension and the task dimension.
Outcome:	The proposed framework improves accuracy on medical evaluations by 2.49% and on financial evaluations 1.2% compared with the second-best method.

Copied to clipboard

Han Wang, Ruiliu Fu, Xuejun Zhang, Jun Zhou, Qingwei Zhao

Challenge:	Existing approaches to stream learning NLP tasks suffer from catastrophic forgetting and are exacerbated when the previous task’s pseudo data is insufficient.
Approach:	They propose to use a new data format to train pseudo questions of previous tasks to stream learning NLP tasks while retaining knowledge of previous ones.
Outcome:	The proposed model is more robust to sufficient and insufficient pseudo-data when the task boundary is both clear and unclear.

Consistent Prototype Learning for Few-Shot Continual Relation Extraction (2023.acl-long)

Copied to clipboard

Xiudi Chen, Hui Wu, Xiaodong Shi

Depth Aware Hierarchical Replay Continual Learning for Knowledge Based Question Answering (2024.lrec-main)

Challenge:	Existing methods for few-shot continual relation extraction are overfitting memory samples, resulting in insufficient activation of old relations and limited ability to handle confusion of similar classes.
Approach:	They propose a few-shot continual relation extraction task that uses memory-enhanced modules to train a model on incrementally few-shot data to avoid forgetting old relations.
Outcome:	The proposed method outperforms existing methods on two commonly-used datasets.

Copied to clipboard

Zhixiong Cao, Hai-Tao Zheng, Yangning Li, Jin Xu, Rongsheng Li, Hong-Gee Kim

Learning Robust Representations for Continual Relation Extraction via Adversarial Class Augmentation (2022.emnlp-main)

Challenge:	Continual learning models adapt well to the latest data but lose ability to remember past data due to changes in the data source.
Approach:	They propose a hierarchical replay framework that allows models to keep a small memory of previous learned data that uses replay.
Outcome:	The proposed model outperforms previous continual learning methods in mitigating catastrophic forgetting.

Copied to clipboard

Peiyi Wang, Yifan Song, Tianyu Liu, Binghuai Lin, Yunbo Cao, Sujian Li, Zhifang Sui

Robustness in NLP

Challenge:	Existing studies attribute catastrophic forgetting to the corruption of the learned representations as new relations come . Continual relation extraction models suffer from catastrophic forgetting when learning new relations .
Approach:	They propose to use adversarial class augmentation mechanism to learn more precise representations . they propose to train the model on a sequence of tasks where two new relations are discovered .
Outcome:	The proposed model improves on two popular benchmarks.

Leitner-Guided Memory Replay for Cross-lingual Continual Learning (2024.naacl-long)

Copied to clipboard

Meryem M’hamdi, Jonathan May

Distilling Causal Effect of Data in Continual Few-shot Relation Learning (2024.lrec-main)

Challenge:	Various continual learning approaches have proposed to mitigate catastrophic forgetting by restricting the data buffer or limiting the data size of a model.
Approach:	They propose to use a human-inspired spaced-repetition technique to prioritize examples for cross-lingual continual learning.
Outcome:	The proposed approach significantly and consistently decreases forgetting while maintaining accuracy across natural language understanding tasks, language orders, and languages.

Copied to clipboard

Weihang Ye, Peng Zhang, Jing Zhang, Hui Gao, Moyao Wang

A Layer-wise Analysis of Supervised Fine-Tuning (2026.acl-long)

Challenge:	Existing methods for learning relational patterns from data are prone to catastrophic forgetting issues due to limited number of samples and continual training mode.
Approach:	They propose a unified causal framework for CFRL to restore causal effects from old data . they establish two additional causal paths from old to predictions by colliding with old data separately in the old feature space.
Outcome:	The proposed method is superior to existing state-of-the-art methods in CFRL task settings.

Copied to clipboard

Qinghua Zhao, Xueling Gong, Xinyu Chen, Zhongfeng Kang, Xinlu Li

Task-Attentive Transformer Architecture for Continual Learning of Vision-and-Language Tasks Using Knowledge Distillation (2023.findings-emnlp)

Challenge:	Existing methods for fine-tuning ignore depth-dependent heterogeneity of instruction-following . a critical gap remains in understanding where these changes occur across the model's depth and which layers are essential for instruction- following.
Approach:	They propose a method which selectively updates critical intermediate layers . they show that effective alignment is architecturally localized rather than distributed .
Outcome:	The proposed method outperforms standard LoRA up to 10.2% on GSM8K with reduced parameter overhead.

Copied to clipboard

Yuliang Cai, Jesse Thomason, Mohammad Rostami

DP-CRE: Continual Relation Extraction via Decoupled Contrastive Learning and Memory Structure Preservation (2024.lrec-main)

Challenge:	Existing algorithms for learning unimodal vision-only or language-only tasks are limited by the size and computational load of fine-tuning large-scale pre-trained neural networks.
Approach:	They propose a transformer-based CL architecture for learning bimodal vision-and-language tasks by increasing the number of the learnable parameters dynamically and using knowledge distillation.
Outcome:	The proposed model reaches state-of-the-art on vision-and-language tasks.

Copied to clipboard

Mengyi Huang, Meng Xiao, Ludi Wang, Yi Du

From Experts to Bases: Orthogonal Subspace Mixture for Continual Multimodal Instruction Tuning (2026.acl-long)

Challenge:	Existing methods for learning relational knowledge are replay-based and prioritize data uniformly . a pronounced bias towards new tasks can be caused by the introduction of new tasks .
Approach:	They propose a framework that decouples the process of prior information preservation and new knowledge acquisition.
Outcome:	Extensive experiments show that the framework outperforms baselines across two datasets.

Copied to clipboard

Pei Chen, Xilai Wang, null Shiqixu, Zejian Li, Lingyun Sun

History repeats: Overcoming catastrophic forgetting for event-centric temporal knowledge graph completion (2023.findings-acl)

Challenge:	Existing parameter-efficient approaches to multimodal Continual Instruction Tuning suffer from knowledge interference and inefficient capacity expansion, limiting scalability.
Approach:	They propose a framework for multimodal Continual instruction tuning that decomposes adaptation weights into a globally shared pool of orthonormal bases to capture task-invariant knowledge.
Outcome:	Experiments show that MoBLoRA outperforms state-of-the-art methods while maintaining superior parameter efficiency.

Copied to clipboard

Mehrnoosh Mirtaheri, Mohammad Rostami, Aram Galstyan

Knowledge Representation

Challenge:	Existing methods for knowledge graph completion are incomplete and can lead to errors . retraining the model with the entire updated TKG can mitigate forgetting but is computationally burdensome.
Approach:	They propose a temporal regularization framework that allows repurposing of parameters . they propose 'clustering-based experience replay' that reinforces the past knowledge .
Outcome:	The proposed framework adapts to new events while reducing catastrophic forgetting.

Preserving Commonsense Knowledge from Pre-trained Language Models via Causal Inference (2023.acl-long)

Copied to clipboard

Junhao Zheng, Qianli Ma, Shengjie Qiu, Yue Wu, Peitian Ma, Junlong Liu, Huawen Feng, Xichen Shang, Haibin Chen

Continual Named Entity Recognition without Catastrophic Forgetting (2023.emnlp-main)

Challenge:	Existing studies attribute catastrophic forgetting to fine-tuning, and they retain pre-trained knowledge indiscriminately without identifying what knowledge is transferable.
Approach:	They propose a unified objective for fine-tuning to retrieve the causality back from pre-trained data and use it to mitigate negative transfer while preserving knowledge.
Outcome:	The proposed method outperforms state-of-the-art fine-tuning methods on commonsense QA datasets and can be implemented as a plug-in module to inflate the performance of existing QA models.

Copied to clipboard

Duzhen Zhang, Wei Cong, Jiahua Dong, Yahan Yu, Xiuyi Chen, Yonggang Zhang, Zhen Fang

Stronger, Lighter, Better: Towards Life-Long Attribute Value Extraction for E-Commerce Products (2024.findings-acl)

Challenge:	Named Entity Recognition (CNER) is a burgeoning area of research . a new paradigm has ushered NER into a non-entity type at the current step t .
Approach:	They propose a pooled feature distillation loss that skillfully navigates the trade-off between retaining knowledge of old entity types and acquiring new ones.
Outcome:	The proposed method outperforms state-of-the-art approaches on ten CNER settings using three datasets.

Copied to clipboard

Tao Zhang, Chenwei Zhang, Xian Li, Jingbo Shang, Hoang Nguyen, Philip Yu

Improving Scheduled Sampling with Elastic Weight Consolidation for Neural Machine Translation (2022.findings-emnlp)

Challenge:	Existing models for attribute value extraction struggle for parameter efficiency and reliability due to data contamination and catastrophic forgetting.
Approach:	They propose to decouple product type and attribute to promote de-contamination and parameter efficiency while scaling up.
Outcome:	The proposed model achieves state-of-the-art performance with affordable parameter size, least historical knowledge forgetting, and greatest robustness against noises.

Copied to clipboard

Michalis Korakakis, Andreas Vlachos

RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World (2026.findings-acl)

Challenge:	Autoregressive models trained with maximum likelihood estimation suffer from exposure bias, i.e. the discrepancy between ground-truth prefixes used during training and model-generated prefix at inference time.
Approach:	They propose to use Elastic Weight Consolidation to better balance mitigating exposure bias with retaining performance.
Outcome:	The proposed method significantly outperforms maximum likelihood estimation and scheduled sampling baselines on four translation datasets.

Copied to clipboard

Hanbing Liu, Lang Cao, Yang Li

Few-shot Table-to-text Generation with Prefix-Controlled Generator (2022.coling-1)

Challenge:	Existing methods to update or supplement large language models struggle under continuous knowledge drift.
Approach:	They propose a dynamic event benchmark and time-aware retrieval baseline that captures how knowledge evolves over time.
Outcome:	The proposed method enables systematic evaluation of model adaptation under continuous knowledge drift.

Copied to clipboard

Yutao Luo, Menghua Lu, Gongshen Liu, Shilin Wang

Text Generation

Challenge:	Neural table-to-text generation approaches are data-hungry and lack labeled data.
Approach:	They propose a prompt-based approach for few-shot table-to-text generation using a task-specific prefix and an input-specific input prefix.
Outcome:	The proposed approach is able to generate table-to-text summaries with a few instances and is validated on human, book and song datasets.

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning (2025.naacl-long)

Copied to clipboard

Elita Lobo, Chirag Agarwal, Himabindu Lakkaraju

Dynamic-prototype Contrastive Fine-tuning for Continual Few-shot Relation Extraction with Unseen Relation Detection (2025.coling-main)

Challenge:	Large language models have emerged as powerful tools for general intelligence, showcasing advanced natural language processing capabilities.
Approach:	They propose to use supervised fine-tuning and Quantized Low-Rank Adapters to improve LLMs' task-specific performance to address privacy and safety risks.
Outcome:	The proposed model improves the accuracy of the chain-of-thought reasonings across four datasets and demonstrates that the faithfulness of CoT reasoning decreases.

Copied to clipboard

Si Miao Zhao, Zhen Tan, Ning Pang, Wei Dong Xiao, Xiang Zhao

Continual Learning with Semi-supervised Contrastive Distillation for Incremental Neural Machine Translation (2024.acl-long)

Challenge:	Existing approaches to learn relations from labeled data overlook task interference in continual learning and memory requirements for different relations.
Approach:	They propose a framework to learn new relations from limited labeled data while preserving knowledge about previously learned relations.
Outcome:	The proposed framework is more practical and comprehensive for real-world scenarios.

Copied to clipboard

Yunlong Liang, Fandong Meng, Jiaan Wang, Jinan Xu, Yufeng Chen, Jie Zhou

FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering (2025.findings-acl)

Challenge:	Multi-domain learning is a good solution for solving domain tasks but it requires retraining when adding a new domain.
Approach:	They propose to exploit unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting.
Outcome:	The proposed framework exploits unlabeled data from the same distributions of the older domains to avoid catastrophic forgetting.

Copied to clipboard

Yichen Li, Zhiting Fan, Ruizhe Chen, Xiaotang Gai, Luqi Gong, Yan Zhang, Zuozhu Liu

SAPT: A Shared Attention Framework for Parameter-Efficient Continual Learning of Large Language Models (2024.acl-long)

Challenge:	Existing prompt-based debiasing methods exhibit instability due to sensitivity to prompt changes . fine-tuning-based techniques incur substantial computational overhead and catastrophic forgetting .
Approach:	They propose a debiasing framework that encodes fairness-related features into separable directions in the hidden activation space.
Outcome:	The proposed framework performs inference-time debiasing without requiring retraining or prompt design . it detects bias signatures in activations and then computes debiased steering vectors . the proposed framework is available to download in the u.s.

Copied to clipboard

Weixiang Zhao, Shilong Wang, Yulin Hu, Yanyan Zhao, Bing Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

Boosting LLM Translation Skills without General Ability Loss via Rationale Distillation (2025.findings-acl)

Challenge:	Existing methods to address catastrophic forgetting and knowledge transfer in large language models (LLMs) ignore potential of aligning the two modules to effectively address catastrophic forgetting and knowledge transfers simultaneously.
Approach:	They propose a Shared Attentive Learning & Selection module to align the PET learning and selection modules to address catastrophic forgetting and knowledge transfer simultaneously.
Outcome:	Experiments on two CL benchmarks show that the proposed framework is superior when scaled to different model sizes, different model architectures and unseen tasks.

Copied to clipboard

Junhong Wu, Yang Zhao, Yangyifan Xu, Bing Liu, Chengqing Zong

SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment (2025.coling-main)

Challenge:	Large Language Models (LLMs) have demonstrated impressive performance across numerous NLP tasks, but fine-tuning them for Machine Translation (MT) often introduces catastrophic forgetting, compromising the broad general abilities of LLMs and introducing potential security risks.
Approach:	They propose a method that harnesses the strong generative capabilities of Large Language Models to create rationales for training data, which are then "replayed" to prevent forgetting.
Outcome:	The proposed approach harnesses the strong generative capabilities of LLMs to create rationales for training data, which are then “replayed” to prevent forgetting.

Copied to clipboard

Yuchun Fan, Yongyu Mu, YiLin Wang, Lei Huang, Junhao Ruan, Bei Li, Tong Xiao, Shujian Huang, Xiaocheng Feng, Jingbo Zhu

LANDeRMT: Dectecting and Routing Language-Aware Neurons for Selectively Finetuning LLMs to Machine Translation (2024.acl-long)

Challenge:	Large language models (LLMs) have demonstrated significant improvements in reasoning abilities, but these improvements are primarily focused on English, leading to inferior performance in non-English scenarios.
Approach:	They propose a multilingual reasoning alignment approach that fine-tunes the layers responsible for multilingual comprehension in one stage.
Outcome:	The proposed method fine-tunes 6 of the 9 layers responsible for multilingual comprehension, while reducing training time by 4.1-11.9 compared to the two-stage method.

Copied to clipboard

Shaolin Zhu, Leiyu Pan, Bo Li, Deyi Xiong

AGTAO: Robust and Stabilized LLM Unlearning via Adversarial Gating Training with Adaptive Orthogonality (2026.findings-acl)

Challenge:	Existing studies have shown promising results in multilingual translation with limited bilingual supervision.
Approach:	They propose a Language-Aware Neuron Detecting and Routing framework that fine tunes LLMs to Machine Translation with diverse translation training data.
Outcome:	The proposed framework selectively finetunes LLMs to MT tasks with diverse translation training data.

Copied to clipboard

Pengyu Li, Lingling Zhang, Zhitao Gao, Yanrui Wu, Yuxuan Dong, Huan Liu, Bifan Wei, Jun Liu

Robustness in NLP

Challenge:	Large Language Models (LLMs) unintentionally memorize sensitive data, posing privacy and security risks.
Approach:	They propose a framework that reconciles unlearning efficacy and utility preservation by using a latent-space gating mechanism to simulate internal recovery attempts.
Outcome:	The proposed framework achieves superior trade-off between unlearning efficacy and model utility.

Improving the Quality Trade-Off for Neural Machine Translation Multi-Domain Adaptation (2021.emnlp-main)

Copied to clipboard

Eva Hasler, Tobias Domhan, Jonay Trenous, Ke Tran, Bill Byrne, Felix Hieber

Continual Few-shot Relation Extraction via Adaptive Gradient Correction and Knowledge Decomposition (2024.findings-acl)

Challenge:	Building neural machine translation systems to perform well on a specific target domain remains a challenge.
Approach:	They propose to train a single NMT system per language pair that performs well across multiple domains.
Outcome:	The proposed approach improves the Pareto frontier on this task.

Copied to clipboard

Jianpeng Hu, Chengxiang Tan, JiaCheng Xu, XiangyunKong XiangyunKong

Orthogonal Subspace Learning for Language Model Continual Learning (2023.findings-emnlp)

Challenge:	Existing methods to learn new relations with limited samples neglect the instability of embeddings in the process of different task training, which leads to catastrophic forgetting.
Approach:	They propose a method to analyze catastrophic forgetting by limiting embedding instability . they propose to decompose knowledge into general and task-related knowledge .
Outcome:	The proposed method outperforms the state-of-the-art model and improves the following degree of embeddings.

Copied to clipboard

Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, Xuanjing Huang

Mixture of LoRA Experts for Continual Information Extraction with LLMs (2025.findings-emnlp)

Challenge:	Existing methods for continual learning in language models suffer catastrophic forgetting when learning sequential tasks.
Approach:	They propose an orthogonal low-rank adaptation approach for continual learning in language models that uses orthogons to learn sequentially.
Outcome:	The proposed approach outperforms state-of-the-art methods on continual learning benchmarks and preserves generalization ability of LLMs on unseen tasks.

Copied to clipboard

Zitao Wang, Xinyi Wang, Wei Hu

Entropy-Based Vocabulary Substitution for Incremental Learning in Multilingual Neural Machine Translation (2022.emnlp-main)

Challenge:	Existing methods to continual information extraction are either task-specialized for a single task or suffer from catastrophic forgetting and insufficient knowledge transfer in continual IE.
Approach:	They propose a continual IE model that uses token-level mixture of LoRA experts with LLMs to extract emerging information across diverse IE tasks incessantly while avoiding forgetting.
Outcome:	The proposed model achieves state-of-the-art performance, effectively mitigating catastrophic forgetting and enhancing knowledge transfer in continual IE.

Copied to clipboard

Kaiyu Huang, Peng Li, Jin Ma, Yang Liu

AnalyticKWS: Towards Exemplar-Free Analytic Class Incremental Learning for Small-footprint Keyword Spotting (2025.findings-acl)

Challenge:	Existing methods to update a multilingual model with new language pairs are expensive and time-consuming.
Approach:	They propose an entropy-based vocabulary substitution method that walks through new language pairs for incremental learning while remaining the size of the original vocabulary.
Outcome:	The proposed method achieves better performance and saves excess overhead in a multilingual machine translation task.

Copied to clipboard

Yang Xiao, Peng Tianyi, Rohan Kumar Das, Yuchen Hu, Huiping Zhuang

Incorporating Syntactic Knowledge into Pre-trained Language Model using Optimization for Overcoming Catastrophic Forgetting (2023.findings-emnlp)

Challenge:	Keyword spotting (KWS) is a useful mechanism to identify spoken commands in voice-enabled systems, but catastrophic forgetting is causing models to lose their ability to recognize earlier keywords.
Approach:	They propose an exemplar-free method that updates model parameters without revisiting earlier data.
Outcome:	The proposed method outperforms existing continual learning methods on a variety of datasets and settings.

Copied to clipboard

Ran Iwamoto, Issei Yoshida, Hiroshi Kanayama, Takuya Ohko, Masayasu Muraoka

Optimizing Lifelong Fine-Tuning for Multiple Tasks via Dataless Distribution Replay (2025.coling-main)

Challenge:	Pre-trained language models lack syntactic knowledge for many tasks that handle complex or long sentences.
Approach:	They propose to use pre-trained language models to incorporate syntactic knowledge into a model by adding additional syntatic knowledge to the model.
Outcome:	The proposed model can be easily applied to downstream tasks that require syntactic knowledge.

Copied to clipboard

Zhenxing Wang

SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment (2025.findings-emnlp)

Challenge:	Existing methods to fine-tune large language models with minimal instruction data are prone to catastrophic forgetting during life-long fine- tuning.
Approach:	They propose a dataless distribution replay approach for life-long fine-tuning that replays the output of the linear layers at previous task stages.
Outcome:	The proposed method achieves significant improve- ments compared to several strong lifelong fine- tuning methods.

Copied to clipboard

Yuqing Huang, Rongyang Zhang, Qimeng Wang, Chengqiang Lu, Yan Gao, null Yiwu, Yao Hu, Xuyang Zhi, Guiquan Liu, Xin Li, Hao Wang, Enhong Chen

Preserving Generalization of Language models in Few-shot Continual Relation Extraction (2024.emnlp-main)

Challenge:	Existing solutions for supervised fine-tuning often lead to catastrophic forgetting, where models lose their previously acquired knowledge and general capabilities.
Approach:	They propose a self-distribution alignment method that aligns input sequence logits to preserve the model’s semantic distribution, thereby mitigating catastrophic forgetting and improving downstream performance.
Outcome:	The proposed method achieves a superior balance between downstream learning and general capability retention.

Copied to clipboard

Quyen Tran, Nguyen Thanh, Nguyen Anh, Nam Hai, Trung Le, Linh Ngo, Thien Nguyen

Improving Continual Few-shot Relation Extraction through Relational Knowledge Distillation and Prototype Augmentation (2024.lrec-main)

Challenge:	Existing methods for Few-shot Continual Relations Extraction (FCRE) are limited in labeled training data and models must learn from a few new samples to solve new tasks.
Approach:	They propose a method that leverages often-discarded language model heads to integrate knowledge from new relations with limited labeled data while avoiding catastrophic forgetting.
Outcome:	The proposed method circumvents catastrophic forgetting and preserves prior knowledge from pre-trained backbones while maintaining accuracy of existing classifications.

Copied to clipboard

Zhiheng Zhang, Daojian Zeng, Xue Bai

DKME: Rethinking Coupled Knowledge Memory for Lifelong Model Editing of Large Language Models (2026.findings-acl)

Challenge:	Existing approaches to Continual Relation Extraction (CRE) are limited in handling the rapid emergence of new relations in real-world scenarios.
Approach:	They propose a framework that integrates prototype-based data augmentation and relational knowledge distillation to solve the problem of Continual Few-shot Relation Extraction (CFRE).
Outcome:	The proposed framework outperforms the state-of-the-art methods on the FewRel and TACRED datasets.

Copied to clipboard

Guanyu Zheng, Wang Zhenyu, He Tingting, Xv Wang, Haochang Wang, Yaokai Huang, Tiejun Zhao

RECALL: REpresentation-aligned Catastrophic-forgetting ALLeviation via Hierarchical Model Merging (2025.emnlp-main)

Challenge:	Existing memory-based editors suffer from catastrophic forgetting as edits accumulate.
Approach:	They propose a method which injects factual updates into large language models without retraining or finetuning into existing memory-based editors.
Outcome:	Experiments on HalluEditBench, CKnowEdit, and WikiDatacounterfact show that the proposed model achieves a more favorable trade-off between editing success and locality compared to baselines while maintaining more stable performance as the edit scale increases.

Copied to clipboard

Bowen Wang, Haiyuan Wan, Liwen Shi, Chen Yang, Peng He, Yue Ma, Haochen Han, Wenhao Li, Tiao Tan, Yongjian Li, Fangming Liu, Gong Yifan, Sheng Zhang

DueT: Image-Text Contrastive Transfer Learning with Dual-adapter Tuning (2023.emnlp-main)

Challenge:	Existing models that require task labels or performance trade-offs are susceptible to catastrophic forgetting.
Approach:	They propose a representation-aware model merging framework for continual learning without access to historical data.
Outcome:	The proposed framework outperforms baselines in knowledge retention and generalization across five NLP tasks and multiple continual learning scenarios.

Copied to clipboard

Taku Hasegawa, Kyosuke Nishida, Koki Maeda, Kuniko Saito

Soft Orthogonal Low-Rank Adaptation for Knowledge Sharing in Large Language Model Continual Learning (2026.acl-long)

Challenge:	Comparative learning models for vision and language models are gaining popularity . dueT trains only adapters inserted into pre-trained image and text encoders .
Approach:	They propose a transfer learning method for vision and language models built by contrastive learning that trains only adapters inserted into the frozen image and text encoders.
Outcome:	The proposed method outperforms fine-tuning, and the LoRA-based adapter method in English and Japanese domains.

Copied to clipboard

Yitong Wang, Xue Han, WenChun Gao, Qian Hu, Jiahui Wang, Ziqing Wang, Lijun Mei, Junlan Feng

FreezeEmpath: Efficient Training for Empathetic Spoken Chatbots with Frozen LLMs (2026.findings-acl)

Challenge:	Existing methods for continual learning (CL) are designed to mitigate catastrophic forgetting while neglecting knowledge sharing across tasks.
Approach:	They propose a framework that facilitates knowledge transfer while mitigating catastrophic forgetting by assigning task-specific parameter subspaces to new tasks . they then leverage attribution scores to evaluate task similarity and employ soft orthogonality between task- specific subspace .
Outcome:	The proposed framework facilitates knowledge transfer while mitigating catastrophic forgetting.

Copied to clipboard

Yun Hong, Yan Zhou, Yang Feng

Natural Language Interfaces

Challenge:	Recent research has made significant progress in developing empathetic spoken chatbots based on large language models (LLMs).
Approach:	They propose an end-to-end empathetic spoken chatbot trained efficiently that generates emotionally expressive speech and outperforms other emmpathetic models in emphatic dialogue, SER, and SpokenQA tasks.
Outcome:	The proposed model outperforms other empathetic models on e-dialog, SER, and SpokenQA tasks and achieves strong results on several speech tasks.

KCTS: Knowledge-Constrained Tree Search Decoding with Token-Level Hallucination Detection (2023.emnlp-main)

Copied to clipboard

Sehyun Choi, Tianqing Fang, Zhaowei Wang, Yangqiu Song

Multi-Task Transfer Matters During Instruction-Tuning (2024.findings-acl)

Challenge:	Existing studies indicate that language models generate non-factual information that is not supported by evidence with a high level of confidence.
Approach:	They propose a knowledge-constrained decoding method that guides a frozen LLM to generate text aligned with the reference knowledge at each decoding step.
Outcome:	The proposed method reduces the risk of misinformation generated by LLMs by reducing training costs and catastrophic forgetting for multi-tasking models.

Copied to clipboard

David Mueller, Mark Dredze, Nicholas Andrews

Safeguard Fine-Tuned LLMs Through Pre- and Post-Tuning Model Merging (2025.findings-emnlp)

Challenge:	Instruction-tuning improves a model’s ability to learn in-context, but the mechanisms that drive in-constext learning are poorly understood.
Approach:	They propose to train a model on hundreds of tasks to improve its ability to learn in-context.
Outcome:	The proposed methods improve model transfer and in-context generalization, suggesting catastrophic forgetting may impact in-constext learning.

Copied to clipboard

Hua Farn, Hsuan Su, Shachi H. Kumar, Saurav Sahay, Shang-Tse Chen, Hung-yi Lee

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning (2024.findings-acl)

Challenge:	Fine-tuning large language models for downstream tasks often leads to catastrophic forgetting, notably degrading the safety of original alignments.
Approach:	They propose to merge the weights of pre- and post-fine-tuned models to improve safety while enhancing performance.
Outcome:	Experiments across different downstream tasks and models validate the method’s practicality and effectiveness.

Copied to clipboard

Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang

Preconditioned Test-Time Adaptation for Out-of-Distribution Debiasing in Narrative Generation (2026.acl-long)

Challenge:	Recent vision-language models (VLMs) have shown impressive capabilities as general visual assistants, but there are two challenges to their performance: (1) lacking task diversity in pretraining and visual instruction tuning; (2) annotation error and bias in GPT-4 synthesized instruction tuning data.
Approach:	They propose a two-stage instruction tuning framework that fine tunes VLMs firstly and further tuned on GPT-4 synthesized data.
Outcome:	The proposed framework outperforms the traditional single-stage visual instruction tuning framework and achieves state-of-the-art performance across a wide range of multi-modal evaluation benchmarks.

Copied to clipboard

Hanwen Shen, Ting Ying, Jiajie Lu, Shanshan Wang

URO-Bench: Towards Comprehensive Evaluation for End-to-End Spoken Dialogue Models (2025.findings-emnlp)

Challenge:	Debiased large language models excel at handling known or low-bias prompts, but fail on unfamiliar and high-biased prompts.
Approach:	They propose a debiasing framework that detects high-bias prompts and triggers context-aware LoRA updates only when a bias-risk score exceeds a threshold.
Outcome:	The proposed framework reduces toxicity/bias score with significantly lower latency than standard optimization methods.

Copied to clipboard

Ruiqi Yan, Xiquan Li, Wenxi Chen, Zhikang Niu, Chen Yang, Ziyang Ma, Kai Yu, Xie Chen

Speech & Audio in NLP

Challenge:	a lack of comprehensive evaluations for SDMs in speech-to-speech (S2S) scenarios is a major challenge for end-to end spoken dialogue models.
Approach:	They propose to provide an extensive evaluation framework for end-to-end spoken dialogue models (SDMs) that includes both cognitive dimensions and paralinguistic cues .
Outcome:	The proposed benchmark is divided into two difficulty levels: basic track and pro track, each comprising 20 test sets, evaluating the spoken dialogue model’s abilities in Understanding, Reasoning, and O**ral conversation.

Making Pre-trained Language Models Better Continual Few-Shot Relation Extractors (2024.lrec-main)

Copied to clipboard

Shengkun Ma, Jiale Han, Yi Liang, Bo Cheng

Rationale-Enhanced Language Models are Better Continual Relation Learners (2023.emnlp-main)

Challenge:	Existing methods to learn new relations with limited labeled data are prone to catastrophic forgetting and overfitting.
Approach:	They propose a framework that uses prompts to acquire more generalized knowledge . they propose CFRE to continuously learn new relations while retaining knowledge of old ones .
Outcome:	The proposed method outperforms state-of-the-art methods by a large margin and significantly mitigates catastrophic forgetting and overfitting in low-resource scenarios.

Copied to clipboard

Weimin Xiong, Yifan Song, Peiyi Wang, Sujian Li

InfoCL: Alleviating Catastrophic Forgetting in Continual Text Classification from An Information Theoretic Perspective (2023.findings-emnlp)

Challenge:	Recent studies have found that catastrophic forgetting arises from the model’s lack of robustness against future analogous relations.
Approach:	They propose a multi-task rationale tuning strategy to help the model learn current relations robustly and conduct contrastive rationale replay to further distinguish analogous relations.
Outcome:	The proposed method outperforms the state-of-the-art models on two benchmarks.

Copied to clipboard

Yifan Song, Peiyi Wang, Weimin Xiong, Dawei Zhu, Tianyu Liu, Zhifang Sui, Sujian Li

MoNMT: Modularly Leveraging Monolingual and Bilingual Knowledge for Neural Machine Translation (2024.lrec-main)

Challenge:	Recent studies have identified the severe performance decrease on analogous classes as a key factor for catastrophic forgetting.
Approach:	They propose a replay-based continual text classification method that uses fast-slow and current-past contrastive learning to perform mutual information maximization and better recover previously learned representations.
Outcome:	The proposed method achieves state-of-the-art on three text classification tasks.

Copied to clipboard

Jianhui Pang, Baosong Yang, Derek F. Wong, Dayiheng Liu, Xiangpeng Wei, Jun Xie, Lidia S. Chao

Causal Intervention-based Few-Shot Named Entity Recognition (2023.findings-emnlp)

Challenge:	Existing models for multi-domain translation tasks only use monolingual data, whereas bilingual data is indispensable for improving the models.
Approach:	They propose a modular strategy that facilitates the cooperation of monolingual and bilingual knowledge in translation tasks by avoiding catastrophic forgetting.
Outcome:	The proposed model exhibits superior generalization and robustness over the conventional approach.

Copied to clipboard

Zhen Yang, Yongbin Liu, Chunping Ouyang

DRAE: Dynamic Retrieval-Augmented Expert Networks for Lifelong Learning and Task Adaptation in Robotics (2025.acl-long)

Challenge:	Existing methods to perform few-shot named entity recognition are limited and overfitting is caused by the spurious correlation resulting from the bias in selecting a few samples.
Approach:	They propose a causal intervention-based few-shot named entity recognition method that blocks the backdoor path between context and label.
Outcome:	The proposed method achieves state-of-the-art in a few-shot named entity recognition (NER) task.

Copied to clipboard

Yayu Long, Kewei Chen, Long Jin, Mingsheng Shang

SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs (2025.acl-long)

Challenge:	Experimental results show that Dynamic Retrieval-Augmented Expert Networks outperforms baseline approaches in long-term task retention and knowledge reuse.
Approach:	They propose a dynamic routing architecture that leverages MoE and Retrieval-Augmented Generation to augment the learning process.
Outcome:	The proposed architecture outperforms baseline approaches in long-term task retention and knowledge reuse.

Copied to clipboard

Yige Xu, Xu Guo, Zhiwei Zeng, Chunyan Miao

FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning (2026.acl-long)

Challenge:	Existing approaches to continuous-space reasoning focus on hard token decoding and suffer from catastrophic forgetting.
Approach:	They propose a method that generates instance-specific soft thought tokens as the initial chain of thoughts and maps them into the LLM’s representation space via a trainable projection module.
Outcome:	The proposed method improves LLM reasoning performance through supervised, parameter-efficient fine-tuning.

Copied to clipboard

Yujie Feng, Hao Wang, Jian Li, Xu Chu, Zhaolu Kang, Yiran Liu, Yasha Wang, Philip S. Yu, Xiao-Ming Wu

Don’t Half-listen: Capturing Key-part Information in Continual Instruction Tuning (2025.acl-long)

Challenge:	Continual learning (CL) for large language models (LLMs) aims to enable sequential knowledge acquisition without catastrophic forgetting.
Approach:	They propose a framework that aligns replay schedules with a model-centric notion of time.
Outcome:	Experiments on three benchmarks show that FOREVER consistently mitigates catastrophic forgetting.

Copied to clipboard

Yongquan He, Wenyuan Zhang, Xuancheng Huang, Peng Zhang, Lingxun Meng, Xiang Zhou, Ke Zeng, Xunliang Cai

ApiQ: Finetuning of 2-Bit Quantized Large Language Model (2024.emnlp-main)

Challenge:	Existing methods to improve instruction tuning for large language models may cause catastrophic forgetting (CF) CF is a problem where previously learned abilities are degraded .
Approach:	They propose a continual instruction tuning method that uses key-part information gain to replay data and refine training objective.
Outcome:	The proposed method achieves superior performance on both seen and held-out tasks.

Copied to clipboard

Baohao Liao, Christian Herold, Shahram Khadivi, Christof Monz

Can Spectral-Clipping Enable Better Learning While Forgetting Less for Low-Rank Adaptation? (2026.acl-long)

Challenge:	Memory-efficient finetuning of large language models (LLMs) has attracted huge attention with the increasing size of LLMs due to the constraints posed by GPU memory limitations and the effectiveness of these methods compared to full finetune.
Approach:	They propose a memory-efficient finetuning framework called ApiQ to restore lost information from quantization by initializing LoRA components and quantizing weights of LLMs.
Outcome:	The proposed framework maintains the original LLM’s activation precision while mitigating error propagation from shallower into deeper layers.

Copied to clipboard

Hyowon Wi, Noseong Park

Unveiling and Addressing Pseudo Forgetting in Large Language Models (2025.findings-acl)

Challenge:	In recent years, low-rank adaptation (LoRA) has emerged as a significant paradigm that freezes pre-trained weights and introduces small, learnable adapters instead of fine-tuning the full set of parameters.
Approach:	They propose a low-rank adaptation approach that injects parameterized singular components with spectral clipping into the pre-trained model.
Outcome:	The proposed method improves performance and retains pre-trained knowledge while preserving the weights of the model.

Copied to clipboard

Huashan Sun, Yizhe Yang, Yinghao Li, Jiawei Li, Yang Gao

Empowering Math Problem Generation and Reasoning for Large Language Model via Synthetic Data based Continual Learning Framework (2025.emnlp-main)

Challenge:	Existing efforts to mitigate catastrophic forgetting in continual learning have not been studied.
Approach:	They propose a rationale-guided replay framework that allows models to leverage their capabilities and provide partial external correct rationales to the original instructions.
Outcome:	The proposed framework mitigates pseudo forgetting while maintaining model plasticity.

Copied to clipboard

Qian Wan, Wangzi Shi, Jintian Feng, Shengyingjie Liu, Luona Wei, Zhicheng Dai, Jianwen Sun

Reasoning

Challenge:	Existing learning frameworks for large language models (LLMs) for math problem generation are limited and lack quality data.
Approach:	They propose a synthetic data based continual learning framework to improve LLMs ability for MPG and math reasoning.
Outcome:	The proposed framework improves performance on large language models and math reasoning using supervised fine-tuning, data synthesis and direct preference optimization.

Harmonizing the Past, Present, and Future: A Null-Space Constrained Region-Specific Method for Continual Learning in LLMs (2026.acl-long)

Copied to clipboard

Jinhui Chen, Shizhu He, Xingchang Yang, Huanxuan Liao, Yequan Wang, Xiangwen Liao, Wenhao Teng, Kang Liu, Jun Zhao

Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning (2026.findings-acl)

Challenge:	Existing continual learning paradigms prioritize instant performance through dense updates, leading to catastrophic forgetting and rapid exhaustion of model capacity.
Approach:	They propose a method that preserves previously acquired knowledge and acquires new task-specific skills while preserving sufficient parameter capacity for subsequent adaptation.
Outcome:	The proposed method is based on the brain's functional partitioning and can be used to map tasks between specialized and generalist neurons.

Copied to clipboard

Anna Borisiuk, Andrey Savchenko, Alexander Panchenko, Elena Tutubalina

Advancing SMoE for Continuous Domain Adaptation of MLLMs: Adaptive Router and Domain-Specific Loss (2025.acl-long)

Challenge:	Existing studies assume that all facts are equally forgettable . popular facts, frequent and widely distributed, may be more deeply embedded than rare ones, making them harder to erase.
Approach:	They propose a benchmark to evaluate how unlearning differs between pretrained and supervised fine-tuned models when fact popularity is taken into account.
Outcome:	The proposed model is compared with pretrained and SFT models on the forget data and shows that it performs better on both models.

Copied to clipboard

Liang Zhang, Ziyao Lu, Fandong Meng, Hui Li, Jie Zhou, Jinsong Su

STAF: Pushing the Boundaries of Test-Time Adaptation towards Practical Noise Scenarios (2024.lrec-main)

Challenge:	Recent studies have explored Continual Instruction Tuning (CIT) in Multimodal Large Language Models (MLLMs), with a primary focus on Task-incremental CIT, where MLLM are required to continuously acquire new tasks.
Approach:	They propose a Sparse Mixture of Expert (SMoE) based method for domain-incremental CIT in Multimodal Large Language Models (MLLMs) . they equip the SMoA module with a domain-specific autoregressive loss (DSAL) they establish a new benchmark to evaluate the efficacy of their method .
Outcome:	The proposed method outperforms all baselines and is based on a Sparse Mixture of Experts (SMoE) module .

Copied to clipboard

Haoyu Xiong, Xinchun Zhang, Leixin Yang, Yu Xiang, Gang Fang

Hierarchical-Task-Aware Multi-modal Mixture of Incremental LoRA Experts for Embodied Continual Learning (2025.acl-long)

Challenge:	Pre-trained language models have demonstrated superior performance on NLP tasks . however, when the training domain and testing domain are taken from different distributions, the deployed model often violates this assumption.
Approach:	They propose a Stable Test-time Adaptation Framework to stabilize the adaptation process.
Outcome:	The proposed framework boosts model robustness to noise distribution shifts while minimizing error accumulation and catastrophic forgetting.

Copied to clipboard

Ziqi Jia, Anmin Wang, Xiaoyang Qu, Xiaowen Yang, Jianzong Wang

Orthogonal Finetuning Made Scalable (2025.emnlp-main)

Challenge:	Existing continual learning setups for embodied intelligence focus on executing low-level actions, neglecting the ability to learn high-level planning and multi-level knowledge.
Approach:	They propose a Hierarchical Embodied Continual Learning Setups (HEC) that divides the agent’s continual learning process into two layers: high-level instructions and low-level actions.
Outcome:	The proposed method reduces the forgetting of old tasks compared to other methods, while orthogonally training the remaining parts.

Copied to clipboard

Zeju Qiu, Weiyang Liu, Adrian Weller, Bernhard Schölkopf

HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning (2026.acl-long)

Challenge:	a recent shift in foundation models has slowed the adoption of finetuning methods . however, its high runtime and memory demands limit its scalability .
Approach:	They propose an input-centric reformulation that uses matrix-vector multiplications instead of cubic multiplication . they extend OFTv2 to support finetuning quantized foundation models and show it outperforms QLoRA .
Outcome:	The proposed model outperforms the popular QLoRA in training stability, efficiency, and memory usage.

Copied to clipboard

Yangfan Wang, Tianyang Sun, Chen Tang, Jie Liu, Wei Cai, Jingchi Jiang

Challenge:	Existing approaches to lifelong model editing apply parameter perturbations to static and dense layers for all instances.
Approach:	They propose a hierarchical reinforcement learning framework that identifies the most knowledge-relevant layers for each editing instance.
Outcome:	The proposed framework boosts the performance of the competitive RLEdit by 8.48% with perturbing only half of the layers per edit.

HiCoLoRA: Addressing Context-Prompt Misalignment via Hierarchical Collaborative LoRA for Zero-Shot DST (2026.findings-acl)

Copied to clipboard

Shuyu Zhang, Yifan Wei, Xinru Wang, Yanmin Zhu, Yangfan He, Yixuan Weng, Yujie Liu, Bin Li